Some Asymptotic Properties of the Support Vector Machine

The support vector machine methodology is a rapidly growing area of research in machine learning. A number of computational learning theoretical type results on the support vector machine have appeared in the machine learning literature. Typically the generalization error of the support vector machine is shown to be bounded by quantities that are related to the empirical margin of the training points or the empirical risk. How the generalization error of the support vector machine compares with the Bayes optimal risk in classification is so far not clear. In this paper we consider the generalization error of a particular form of the support vector machine. We show that this support vector machine approaches the Bayes optimal rule in a direct fashion, and its generalization error converges quickly to the Bayes optimal risk. The results are established under general conditions allowing discontinuity in the underlying class probability function. The results clarify the advantages and limitations of the support vector machine, and give insights on how the support vector machine can be extended systematically.